Evaluating a clinical decision support point of care instrument in low resource setting

Background Clinical pathways are one of the main tools to manage the health care’s quality and concerned with the standardization of care processes. They have been used to help frontline healthcare workers by presenting summarized evidence and generating clinical workflows involving a series of tasks performed by various people within and between work environments to deliver care. Integrating clinical pathways into Clinical Decision Support Systems (CDSSs) is a common practice today. However, in a low-resource setting (LRS), this kind of decision support systems is often not readily accessible or even not available. To fill this gap, we developed a computer aided CDSS that swiftly identifies which cases require a referral and which ones may be managed locally. The computer aided CDSS is designed primarily for use in primary care settings for maternal and childcare services, namely for pregnant patients, antenatal and postnatal care. The purpose of this paper is to assess the user acceptance of the computer aided CDSS at the point of care in LRSs. Methods For evaluation, we used a total of 22 parameters structured in to six major categories, namely “ease of use, system quality, information quality, decision changes, process changes, and user acceptance.” Based on these parameters, the caregivers from Jimma Health Center's Maternal and Child Health Service Unit evaluated the acceptability of a computer aided CDSS. The respondents were asked to express their level of agreement using 22 parameters in a think-aloud approach. The evaluation was conducted in the caregiver's spare-time after the clinical decision. It was based on eighteen cases over the course of two days. The respondents were then asked to score their level of agreement with some statements on a five-point scale: strongly disagree, disagree, neutral, agree, and strongly agree. Results The CDSS received a favorable agreement score in all six categories by obtaining primarily strongly agree and agree responses. In contrast, a follow-up interview revealed a variety of reasons for disagreement based on the neutral, disagree, and strongly disagree responses. Conclusions Though the study had a positive outcome, it was limited to the Jimma Health Center Maternal and Childcare Unit, and hence a wider scale evaluation and longitudinal measurements, including computer aided CDSS usage frequency, speed of operation and impact on intervention time are needed. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-023-02144-0.


EVALUATION DESIGN
The CDS system was developed and deployed on a Raspberry Pi 4 Model B, which has a quad-core 64-bit processor and 4GB of RAM. The CDS instrument WEB-APP is accessed via a smart phone's mobile data (or wireless network). The purpose of the system was to promote high-quality care and assist healthcare workers in identifying referral and locally treatable cases, as well as patient care activities.
The framework for evaluating an artificial intelligence-enabled clinical decision support system was developed based on Ji, Mengting, et al.2021 and has been customized to our needs. We adopted 22 of the 28 parameters from the evaluation framework. Outcome 1 changes, service quality, and productivity (related to process change) will not be considered because the CDS instrument will be evaluated after the clinical decision is made. Furthermore, since the variables "information satisfaction, service, and system's quality" were difficult to distinguish, we aggregated them as overall quality.
To set up an experiment for evaluation, the following activities will be carried out: The screening for participant eligibility and contacting the person of interest, as well as setting up meeting specifics such as time and location, will take place. To provide for as much participant flexibility as feasible, evaluation will be done on participant spare-time, because the number of health professionals at the health center's maternal and child healthcare unit is limited (approximately five to eight) and they are too busy to complete their ordinary daily activities. As a result, instead of an instant patient-by-patient evaluation, the evaluation will be completed over the course of a half-day.
After clarifying the goal and obtaining consent, the participants will be given a guide with detailed step-by-step instructions how to use WEB-APP to assist them in better preparing for the activity. Ji, M., Genchev, G. Z., Huang, H., Xu, T., Lu, H., & Yu, G. (2021)

. Evaluation Framework for Successful Artificial Intelligence-Enabled Clinical Decision Support Systems: Mixed Methods Study. Journal
The think-aloud protocol will be followed while the participant attempts to use WEB-APP. The system will be evaluated after the clinical decision was made using the concurrent think-aloud approach. In a thinking aloud test (TA-Test), participants are asked to use the CDS instrument while continuously thinking out loud 1 .
In the end, the participant will complete the questionnaire (found on pages 3 and 4 of this document). It is a form of psychometric response scale in which respondents express their level of agreement to a statement in five scores: (1) strongly disagree; (2) disagree; 2 (3) neutral; (4) agree; and (5) strongly agree. The questionnaire is structured into five sections with a total of 22 questions to validate and measure the instrument's characteristics in the following order: ease of use (6/22), system quality (2/22), information quality (2/22), decision changes (2/22), process changes (5/22), and user acceptance (5/22).
The questionnaire was translated into Amharic. A freelance and experienced translator then reviewed the translated questionnaire to resolve any discrepancies between the original English version and the translated Amharic questionnaire. The questionnaire will be accessible for submission through mobile, laptop, or paper-based format. We prefer mobile or laptop-based formats to paper-based formats, unless in exceptional instances.
In conclusion, the findings will be analyzed to gain insights and uncover common patterns in order to identify future actions.
Moreover, personal information exclusively used for questionnaire verification did not appear in reporting or results. In general, we are committed to protecting your personal information and respecting your privacy.

GENERAL INSTRUCTIONS FOR FILLING THE QUESTIONNAIRE
The information you provide will be used solely to test and evaluate the WEB-APP CDS instrument. The data will be valuable in improving the instrument.
The following are statements for evaluating CDS POC in LRS on which some people agree and others disagree. We would like to indicate your opinion after each statement by putting an "X" in the box that best indicates the extent to which you strongly disagree, disagree, neutral, agree, or strongly disagree. Furthermore, when the participant's response is "strongly disagree, disagree, or neutral", additional explanations will be requested, which will be reviewed later for further WEB-APP improvement. The information may be found in the column labeled "comment in the case of neutral, disagree, and strongly disagree".